Crowdsourcing Thumbnail Captions: Data Collection and Validation

Speech interfaces, such as personal assistants and screen readers, read image captions to users—but typically only one caption is available per image, which may not be adequate for all situations (e.g., browsing large quantities of images). Long provide a deeper understanding an but require more time listen to, whereas shorter allow thorough comprehension, yet have the advantage being faster consume. We explore how effectively collect both thumbnail captions—succinct descriptions meant consumed quickly—and comprehensive captions—which individuals understand visual content in greater detail; we consider text-based instructions time-constrained methods at these two levels detail find that method most effective collecting while preserving accuracy. Additionally, verify authors using this are still able focus on important regions by tracking their eye gaze. evaluate our collected along human-rated axes—correctness, fluency, amount detail, mentions concepts—and discuss potential model-based metrics perform large-scale automatic evaluations future.